databricks: support Lakehouse//RT, auto-detecting the SEA backend - #9879
Conversation
Lakehouse//RT warehouses only speak the Statement Execution API (SEA) and reject the Thrift protocol, so Rill (pinned to databricks-sql-go v1.10.0) could not connect to them at all. Bump databricks-sql-go v1.10.0 -> v1.15.1 (adds the SEA/kernel backend) and auto-detect RT: on first connect the connector probes once and, if the warehouse rejects Thrift, transparently switches to the SEA backend. The decision is cached and shared by both the OLAP path and the warehouse ingest path (effectiveDSN), so an RT warehouse works with no configuration. DBSQL warehouses accept Thrift and are unchanged. A `use_kernel` connector property (default false) is available as an explicit override. The SEA backend exports Arrow C Data rather than IPC streams, so the bulk ingest path in warehouse.go falls back from GetArrowIPCStreams to GetArrowBatches (re-serialized to a self-contained Arrow IPC stream via the driver's Arrow v12 writer, which the existing v18 ipc.Reader consumes) when the driver returns ErrNotSupportedByKernel. The OLAP path already worked over SEA. information_schema.Lookup previously JOINed information_schema.tables and columns; that join forces a shuffle that RT's Photon rejects (PHOTON_INTERNAL_ERROR, retry unsupported), breaking the schema browser. Split it into two filtered point-lookups (no shuffle); equivalent on DBSQL. Adds unit tests for DSN resolution, the RT-detection predicate, and the useKernel DSN helper, and documents the behavior. Co-authored-by: Isaac <no-reply@databricks.com>
# Conflicts: # go.mod # go.sum
Adds TestOLAP_LakehouseRT, a live integration test that drives the full Rill Databricks OLAP path against a Lakehouse//RT (Reyden) SQL warehouse over the auto-detected SEA backend. The connector config passes only the DSN (no use_kernel), so a passing query proves the Thrift->SEA autodetection end to end. Because Lakehouse//RT speaks only SEA, the test requires the SEA-via-kernel backend and carries //go:build databricks_kernel (CGO), excluding it from the default Thrift-only build and the standard `go test -short ./...` CI job. It is otherwise gated like the existing Databricks/Snowflake live tests: a leading t.Skip disables it by default, testmode.Expensive keeps it out of normal runs, and it needs RILL_RUNTIME_DATABRICKS_RT_TEST_DSN to point at an RT warehouse. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
nishantmonu51
added
Type:Feature
|
Hey @dey-abhishek Thanks for the PR. Statically linking the platform specific binary is not great. Ideally it would have much better if the Go driver like the JDBC and Python driver implemented a pure Go based SEA backend instead of relying on Rust bindings. For now we will link the kernel bindings statically. I made the required changes and simplified some logic as well. Also I don't have access to Lakehouse RT so just to confirm were you able to do any live testing with the new backend and did you run the tests in file |
|
Thanks @k-anshul! Confirming live testing on this branch, including your static-bindings and simplification changes: End-to-end via the Rill UI — built with
DBSQL (Thrift) behavior is unchanged and was re-verified as well. |
|
Thanks for confirming @dey-abhishek |
There was a problem hiding this comment.
1. A transient probe failure permanently poisons the connector — runtime/drivers/databricks/databricks.go, backendDSN
c.dbErr is sticky (getDB short-circuits on it, and backendDSN returns it on entry). Previously it could only be set by sqlx.Open failing, which is deterministic. Now a live PingContext sets it. Databricks warehouses auto-stop, so the first connect after idle starts the warehouse and can time out or return a transient 5xx — after which every subsequent query returns the stale error until the instance is reopened. The ctx.Err() == nil guard does not cover server-side timeouts or network errors. Cache the RT/no-RT decision, not the failure.
2. Every DBSQL connection now pays an extra connect round-trip — same function
backendDSN opens a second pool and pings it before the real pool, for everyone who has not set use_kernel; Ping() then pings again through getDB. On a cold warehouse this doubles the cold-start path. Probing lazily — retry once on a Thrift rejection at first query, reusing rtRequiresSEA — gets the same auto-detection at zero cost for DBSQL.
3. -tags databricks_kernel is enabled unconditionally, which the description says is a follow-up — Makefile:7, Makefile:11, .github/workflows/cli-release.yml:73, .github/workflows/rill-cloud.yml:99-101
The body says enabling the tag in shipped binaries is "left as a follow-up for maintainers", but the diff enables it for the released CLI, the cloud image, and local make cli / make cli-only. By the PR's own numbers that is ~60–95 MB of native archive per platform added to the CLI download and to every local dev build. Either drop these four hunks and keep the graceful-error path, or raise the size decision explicitly.
4. olap_rt_test.go:23 is dead code
The unconditional t.Skip makes the whole function unreachable, including via the invocation documented directly above it. The build tag and the RILL_RUNTIME_DATABRICKS_RT_TEST_DSN check already gate it; remove the t.Skip.
5. Lookup now returns ErrNotFound for a missing table — runtime/drivers/databricks/information_schema.go:161
Previously a missing table produced an OlapTable with an empty schema. ErrNotFound looks like the right contract, but it is a behavior change unrelated to the RT fix and is not mentioned in the description. Worth confirming callers handle it (AllFromInformationSchema, schema browser).
|
1. A transient probe failure permanently poisons the connector — Yes this is expected. This is what happens in normal connector reconcile as well and the user is expected to refresh the connector. 2. Every DBSQL connection now pays an extra connect round-trip — same function This is acceptable trade off. We already do a ping and pings are expected to fast.
No it does not. The first ping will wake up the warehouse which should make second ping fast. If the warehouse does not wake up then it won't try second ping.
This requires meaningful code changes and is not very clean. If we don't want to pay the cost then I think better alternative is to just let the user forcefully select it but it will remove the nice auto detection functionality. 3. A decision was made to ship the SDK statically. The new binary is ~10 MB larger than v0.89.4 across all four platforms (~6–8% growth). (zipped size) 4. This is expected. We neither have a permanently running databricks test cluster nor databricks RT cluster so it is expected to spin a cluster when making changes and test accordingly. 5. This is an expected contract and the earlier change was a bug. |
3 participants
databricks: support Lakehouse//RT, auto-detecting the SEA backend (backward compatible with DBSQL)
Problem
Lakehouse//RT (real-time) Databricks warehouses only speak the Statement Execution
API (SEA) protocol and reject the Thrift/HiveServer2 protocol that
databricks-sql-gouses by default:
Rill currently pins
databricks-sql-go v1.10.0, which predates the driver's SEA/kernelbackend, so Rill cannot connect to Lakehouse//RT warehouses at all — the connector fails
at open, before any query runs.
What this changes
databricks-sql-gov1.10.0 → v1.15.1. v1.15.0 introduced the opt-inSEA/kernel backend (
useKernel=true/WithUseKernel(true)). The default remainsThrift, so this bump alone changes no behavior.
connector probes once; if the warehouse rejects Thrift (the RT signal), it
transparently switches to the SEA backend. The probe result is cached and shared by
both the OLAP path and the warehouse ingest path (
effectiveDSN). DBSQL warehousesaccept Thrift and are left unchanged. This means a user can add an RT warehouse in
the UI with no special knowledge — no
use_kernel, noUseThriftClient— and itjust works.
use_kernelconnector property (defaultfalse) as an explicit override. ForcesSEA and skips the probe. Rarely needed given auto-detection, but useful to force SEA.
warehouse.goingest fallback. The bulk-ingest path prefersGetArrowIPCStreams(unchanged Thrift path). The SEA backend doesn't implement IPCstreams — it exports Arrow C Data — and returns an error wrapping
dbsqlerr.ErrNotSupportedByKernel. On that sentinel we fall back toGetArrowBatchesand re-serialize each record to a self-contained Arrow IPC stream(via the driver's Arrow v12
ipc.Writer), which the existing v18ipc.Reader→parquet path consumes untouched. This bridges the driver's Arrow v12 ↔ Rill's Arrow
v18 without changing the DBSQL path. Rill's OLAP path (
olap.go,sqlxrows) wasalready SEA-compatible and is unchanged.
information_schema.Lookupde-JOINed. The per-table schema lookup JOINedinformation_schema.tablesandcolumns; that join forces a shuffle that RT's Photonrejects (
PHOTON_INTERNAL_ERROR, retry unsupported), breaking the schema browser.Split into two filtered point-lookups (no shuffle) — verified against a real RT
warehouse (the JOIN fails, both point-lookups succeed); equivalent on DBSQL.
Backward compatibility
databricks_kerneltag) compile unchanged;use_kerneldefaults tofalse, so DBSQL warehouses are byte-for-byte on the same Thrift path.use_kernel: trueis set in a build without the kernel backend linked, the driverfails at connect with a clear error wrapping
ErrKernelNotCompiled(no silent Thriftfallback, no crash).
Build / distribution note (needs a maintainer decision)
The SEA backend links a native library and is gated behind
-tags databricks_kernelwith CGO. Rill already builds with CGO enabled (DuckDB, confluent-kafka), so the only
additional cost is adding the tag and linking the per-platform kernel archive
(
databricks-sql-kernel-bindings, ~60–95 MB/platform). This PR makes RT supportavailable and correct when built with the tag, and a graceful error otherwise. How
(and whether) to enable the tag in shipped release binaries is left as a follow-up for
maintainers.
It was decided to link the library statically. The overall size increase was about 20-25MB.
Testing
runtime/drivers/databricks/dsn_internal_test.go:TestResolveDSN(backend selection — no
useKernelby default, set when opted in, DSN pass-through,?/&joining, no duplicate),TestRTRequiresSEA(only the Thrift-not-supportedmessage triggers the switch — 403/refused/nil don't), and
TestWithUseKernel.use_kernelset: the connector logged the Thrift→SEA switch and both the connectorand a model reconciled; a regular DBSQL warehouse stayed on Thrift and reconciled.
use_kernel: true): connector + aSELECT current_catalog(), now()model reconcile successfully and materialize into DuckDB.
use_kernelunset, Thrift): aSELECT current_catalog(), current_version(), now()model reconciles successfully —current_version()works here but isUNRESOLVED_ROUTINEon RT, confirming theThrift path is unaffected.
Checklist: